Product design, configuration and decision system using machine learning

ABSTRACT

A product configuration design system, includes a product configuration design server, including a processor, a non-transitory memory, an input/output, a product storage, a configuration library, and a machine learner; and a product configuration design device, which enables a user to select a three-dimensional object representation, a collection, and an inspiration source, such that the product configuration design server generates a plurality of product configurations as an output from a machine learning calculation on a configuration generation model, which takes as input the three-dimensional object representation, the collection, and the inspiration source. Also disclosed is a method of selecting a three-dimensional object representation, a collection, and an inspirations source; and generating product configurations.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.62/797,073, filed Jan. 25, 2019; which is hereby incorporated herein byreference in its entirety.

FIELD OF THE INVENTION

The present invention relates generally to the field ofthree-dimensional product design, and more particularly to methods andsystems for machine-learning and artificial intelligence based productdesign using a set of selected colors and materials for the creation ofmaterial and color combinations on a 3D object, driven by input fromstored personal images and content collections, external social media,photo libraries, and other containers of personal visual information.

BACKGROUND OF THE INVENTION

The creation of visual representations of 3D objects along with variousconfigurations using different color and material combinations has beenaround for quite some time. From a consumer facing standpoint, manyproducts nowadays can be configured through an online configurator. Onthe design side during the product development process, the concept ofconfigurators enables designers to present and make design decisionswhile being able to visualize the product in a photorealisticrepresentation under 3D lighting conditions, even with the use of VR andAR has been explored for several years.

When looking at using configurators as part of the actual designprocess, in particular through the exploration phase of the designprocess, only few solutions are available and of those that are beingoffered, the user experience is lacking when it comes to setting upconfigurations. Furthermore, this process is largely driven by manualinteraction, pulling inspiration from many sources stored in variousmedia, and is limited by the amount of time and resources the designerhas available to explore many alternatives. Any design decisions thathave been made on color and material configurations are stored in atraditional way, either digitally or manually in form of print orphysical sample.

However, while these decisions serve as references and as such can aidas input for the next design cycle, there is currently no smart way ofreusing past design decisions and being able to combine this efficientlywith trend and mood data captured in digital or yet again physical form.All of this input relies on human interaction, requiring for anindividual to analyze the data and translate that into next generationdesign proposals.

As such, considering the foregoing, it may be appreciated that therecontinues to be a need for novel and improved devices and methods forpresenting digital representations of materials via methods of productvisualization.

SUMMARY OF THE INVENTION

The foregoing needs are met, to a great extent, by the presentinvention, wherein in aspects of this invention, enhancements areprovided to the existing model of product visualization.

In an aspect, the product configuration design system allows designersto use Artificial Intelligence (AI) and Machine Learning (ML) fordesigning and configuring materials and colors on 3D objects, whileteaching the system to make better and more targeted design decisions inthe future.

In a related aspect, the product configuration design system can usemachine learning/AI to help designers to parse/scan/crawl largedatabases of materials and colors based on personal mood and trendboards, and personal or public collections of images and photos onsocial media sites such as PININTEREST™, INSTAGRAM™ and FACEBOOK™. AIwill present the designer with potentially large amount of combinationswhich is primarily driven by the amount of input the application candraw from.

In related aspects, the product configuration design system provides anintuitive decision-making system, which through simple left or rightswiping, or tapping, allows the designer to keep or dismissconfigurations. Machine Learning will be able to smartly evolve for anarrowed, more targeted offerings of design configurations. MachineLearning is further taught by input from other people with whom designis being shared with, allowing not only to capture the configurationitself, but also demographics, gender, age and potentially more. Oncethe machine has acquired critical mass, it will then allow designers touse AI to further drive the design with additional input parametersincluding age, gender, and demographics. The result are not staticimages, but rather three-dimensional, configured objects that can beviewed interactively in a 3D viewer, as well as AR, under variouslighting conditions, to give a photorealistic representation of thedesign.

In another related aspect, design configurations can be further refinedthrough manual interaction by locking certain parts of the object toprevent changes, as well as visualized with additional design elementssuch as graphic print and trim.

There has thus been outlined, rather broadly, certain embodiments of theinvention in order that the detailed description thereof herein may bebetter understood, and in order that the present contribution to the artmay be better appreciated. There are, of course, additional embodimentsof the invention that will be described below and which will form thesubject matter of the claims appended hereto.

In this respect, before explaining at least one embodiment of theinvention in detail, it is to be understood that the invention is notlimited in its application to the details of construction and to thearrangements of the components set forth in the following description orillustrated in the drawings. The invention is capable of embodiments inaddition to those described and of being practiced and carried out invarious ways. In addition, it is to be understood that the phraseologyand terminology employed herein, as well as the abstract, are for thepurpose of description and should not be regarded as limiting.

As such, those skilled in the art will appreciate that the conceptionupon which this disclosure is based may readily be utilized as a basisfor the designing of other structures, methods and systems for carryingout the several purposes of the present invention. It is important,therefore, that the claims be regarded as including such equivalentconstructions insofar as they do not depart from the spirit and scope ofthe present invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram illustrating a product configurationdesign system, according to an embodiment of the invention.

FIG. 2 is a schematic diagram illustrating a product configurationdesign device, according to an embodiment of the invention.

FIG. 3 is a schematic diagram illustrating a product configurationdesign server, according to an embodiment of the invention.

FIG. 4A is an illustration of physical material samples of the productconfiguration design system, according to an embodiment of theinvention.

FIG. 4B is an illustration of a digital material representation of theproduct configuration design device, according to an embodiment of theinvention.

FIG. 4C is an illustration of a three-dimensional digital object of theproduct configuration design device, according to an embodiment of theinvention.

FIG. 5 is an illustration of a graphical user interface of the productconfiguration design device, according to an embodiment of theinvention.

FIG. 6 is an illustration of a graphical user interface of the productconfiguration design device, according to an embodiment of theinvention.

FIG. 7 is an illustration of a graphical user interface of the productconfiguration design device, according to an embodiment of theinvention.

FIG. 8 is an illustration of a graphical user interface of the productconfiguration design device, according to an embodiment of theinvention.

FIG. 9 is an illustration of a graphical user interface of the productconfiguration design device, according to an embodiment of theinvention.

FIG. 10 is an illustration of a graphical user interface of the productconfiguration design device, according to an embodiment of theinvention.

FIG. 11 is an illustration of a graphical user interface of the productconfiguration design device, according to an embodiment of theinvention.

FIG. 12 is an illustration of a graphical user interface of the productconfiguration design device, according to an embodiment of theinvention.

FIG. 13 is an illustration of a graphical user interface of the productconfiguration design device, according to an embodiment of theinvention.

FIG. 14 is an illustration of a graphical user interface of the productconfiguration design device, according to an embodiment of theinvention.

FIG. 15 is an illustration of a graphical user interface of the productconfiguration design device, according to an embodiment of theinvention.

FIG. 16 is a flowchart illustrating steps that may be followed, inaccordance with one embodiment of a method or process of productvisualization.

DETAILED DESCRIPTION

Before describing the invention in detail, it should be observed thatthe present invention resides primarily in a novel and non-obviouscombination of elements and process steps. So as not to obscure thedisclosure with details that will readily be apparent to those skilledin the art, certain conventional elements and steps have been presentedwith lesser detail, while the drawings and specification describe ingreater detail other elements and steps pertinent to understanding theinvention.

The following embodiments are not intended to define limits as to thestructure or method of the invention, but only to provide exemplaryconstructions. The embodiments are permissive rather than mandatory andillustrative rather than exhaustive.

In the following, we describe the structure of an embodiment of aproduct configuration design system 100, with reference to FIG. 1, insuch manner that like reference numerals refer to like componentsthroughout; a convention that we shall employ for the remainder of thisspecification.

In various embodiments, the product configuration design system 100provides a novel way of automatically configuring materials and colorcombinations on a 3D product from a cloud-based material library syncedto a local device that is being fed through an AI engine that looks atpersonal mood boards stored inside the application, and imagecollections saved in social media applications. These AI generatedconfigurations can be either fully automated or follow the input of theuser who can determine which parts of the 3D object are beingautomatically configured. AI can further be used to apply materials in asmart way by processing material properties and tags, as well as theobject's layers, to determine which type of material goes to what partof the object. All resulting configurations can be viewed interactivelyin 3D under real world-lighting conditions, as well as AR. Upon reviewof the AI generated results by the user inside the app and by sharingwith people outside the organization users can accept or reject thepresented combinations. This will teach the system through MachineLearning and as such make the AI engine smarter when it comes to theconfiguration of future products. This will result in capturing andanalyzing trend data, not just based on color and materials, but alsobased on gender, age, demographics and more, allowing for moretarget-driven design in the future.

In other related embodiments, a product configuration design systemautomatically configures materials and color combinations on athree-dimensional object from a material library provided by a machinelearning engine that parses personal mood boards stored in the device,and image collections saved in social media applications. These AIgenerated configurations can be either fully automated or follow theinput of the user who can determine which parts of the 3D object arebeing automatically configured. AI can further be used to applymaterials in a smart way to by processing material properties and tags,as well as the object's layers, to determine which type of material goesto what part of the object. AI generated results can be accepted orrejected, such that machine learning is used to train the system,resulting in capturing and analyzing trend data, based on color andmaterials, gender, age, demographics, allowing for target-driven designof future collections.

In an embodiment, as shown in FIG. 1, a product configuration designsystem 100 can include:

-   -   a) a product configuration design server 102, comprising a        configuration generation model 316; and    -   b) a product configuration design device 104, such that the        product configuration design device 104 can be connected to the        product configuration design server 102;    -   wherein the product configuration design device 104 is        configured to enable a user to select a three-dimensional object        representation 400 c, a collection 610, and an inspiration        source 710;    -   such that the product configuration design server 102 generates        a plurality of product configurations 810 as an output from a        machine learning calculation on the configuration generation        model 316, which takes as input the three-dimensional object        representation 400 c, the collection 610, and the inspiration        source 710.

In a related embodiment, as shown in FIG. 2, a product configurationdesign device 104 can include:

-   -   a) a processor 202;    -   b) a non-transitory memory 204;    -   c) an input/output 206;    -   d) a camera 207;    -   e) a screen 208;    -   f) a product visualizer 210;    -   g) an image library 212; and    -   h) a product editor 214; all connected via    -   i) a data bus 220.

In a related embodiment, a product configuration design server 102 caninclude:

-   -   a) a processor 302;    -   b) a non-transitory memory 304;    -   c) an input/output component 306;    -   d) a product storage 310, for storing a customizable tagging        system, comprising a hierarchy of tags, such that each tag is        associated with at least one three-dimensional object;    -   e) a configuration library 312, for storing product        configurations; and    -   f) a machine learner 314, which uses/processes a machine        learning algorithm for training and executing a configuration        generation model 316. The machine learning algorithm can use        well-known methods of machine learning, including artificial        neural networks, such as deep learning feed-forward neural        networks with back-propagation learning, genetic algorithms;        support vector machines, and cluster classification; all        connected via    -   g) a data bus 320.

In a further related embodiment, the machine learning algorithm can be aconvolutional artificial neural network with at least two hidden layers,such that the configuration generation model is implemented by theoptimized/trained convolutional artificial neural network, which can betrained/optimized using well-known artificial neural network deeplearning methods, including backpropagation and other non-linearfunction optimization methods. In many related embodiments,learning/training can be unsupervised, in order to ensure hidden/unknownrating bias is found/incorporated, but in some embodiments, learning maybe supervised or partially supervised, and may employ reinforcementlearning.

In related embodiments, the product configuration design device 104 caninclude configurations as:

-   -   a) a mobile app, executing on a mobile device, such as for        example an ANDROID™ or IPHONE™, or any wearable mobile device;    -   b) a tablet app, executing on a tablet device, such as for        example an ANDROID™ or IOS™ tablet device;    -   c) a web application, executing in a Web browser;    -   d) a desktop application, executing on a personal computer, or        similar device; or    -   e) an embedded application, executing on a processing device,        such as for example a smart TV, a game console or other system.

It shall be understood that an executing instance of an embodiment ofthe product configuration design system 100, as shown in FIG. 1, caninclude a plurality of product configuration design devices 104, whichare each tied to one or more users 122.

An executing instance of an embodiment of the product configurationdesign system 100, as shown in FIG. 1, can similarly include a pluralityof product configuration design servers 102.

In related embodiments of the product configuration design system 100,functions provided by the product configuration design device 104 incommunication with the product configuration design server 102 will bedisclosed in the following with reference to particular applicationviews of the graphical user interface of the product configurationdesign device 104.

In further related embodiments, as seen in the graphical user interfaceillustrated, the user 122 can be guided along the setup process throughicons in the bottom tool bar, as for example seen in FIG. 5. As soon asthe current step is satisfied the next step will become available. Thiswill be indicated by the change in the color of the icon of the nextstep. Use of Artificial intelligence and Machine Learning is an integralpart of the product configuration design system 100 and will describedwhere applicable.

In further related embodiments, in order to facilitate use of AI/machinelearning, such that layers or portions of 3D objects 400 c, as shown inFIGS. 4A, 4B, and 4C, can be either named or tagged accordingly, suchthat digital material representations 400 b corresponding to physicalsamples 400 a can be mapped to the individual layers, and end resultscan be subject to machine learning using the corresponding tags. Digitalmaterial representations 400 b can be tagged to indicate the use of thematerial in the 3D object. This way the generated configurations are“smart” configurations from the beginning and can aid in MachineLearning. AI and ML may also use visual analysis to automaticallyrecognize where to place materials & graphics.

In a related embodiment, as shown in FIG. 5, the product editor 214 canbe configured to provide a graphical user interface 500 for selecting athree-dimensional object 510, as a first step in a product configurationprocess. The toolbar 550 on the bottom guides the user along in theprocess. The product editor 214 can be configured to show the 3D objects510 that are available to be used for configuration.

In a further related embodiment, the product editor 214 can beconfigured to start the process of creating configurations driven byArtificial Intelligence (AI), based on an initial 3D model 400 c 510.The 3D model 400 c 510 can include individual layers 410 or portions 410that can be configured. The 3D objects 400 c can be stored locally onthe product configuration design device 104, and can be retrieved,uploaded, synchronized in communication with the product configurationdesign server 102, which can be cloud-based. The user browses throughall available assets and selects the object 400 c with which s/he wantsto work. In order to accept an asset 510, the user 122 swipes left, inorder to reject and therefore remove an asset from a collection, theuser swipes right. Once the 3D asset 510 is selected, the next step willbecome available.

In a related embodiment, as shown in FIG. 6, the product editor 214 canbe configured to provide a graphical user interface 600 for displayingand selecting the collections 610 that are available to be used for theAI driven configurations. Collections contain materials, colors andgraphics. All configurations can be synced with the productconfiguration design server 102 (i.e. online platform/backend) andcopied locally to the product configuration design device 104.

In a further related embodiment, collections 610 can include materialsand colors, for example for the next season. These collections 610 areusually prepared by a team of designers and can be as large as the userchooses. A collection 610 can also include an entire online collectionfrom an approved vendor in combination with any color palette (such asPANTONE™ Color or other color schemes) as input for the application todraw from when creating configurations. Alternatively, the selection ofmaterials can be instigated directly from the inspiration content fedinto the AI & ML processing, provided by the machine learning component314 of the product configuration design server 102.

In a related embodiment, a collection 610 can include a plurality ofmaterials 400 b and a plurality of colors.

In another related embodiment, as shown in FIG. 7, the product editor214 can be configured to provide a graphical user interface 700 forselecting the inspiration source 710, as a third step in the setup. Theuser selects the inspiration source 710, which should be used to createthe configurations using the materials and colors as selected in thegraphical user interface 600 for displaying and selecting thecollections 610.

In a further related embodiment, the inspiration source 710, as shown inFIG. 7, can come from various sources 710. The machine learningcomponent 314 of the product configuration design server 102 can beconfigured to process the select source, which may be a custom usercollection often described as a “mood” or “trend” board, or other onlinecollections of images such as PININTEREST™, INSTAGRAM™, FACEBOOK™ andmore. The machine learning component 314 will take the individual orpublic boards, depending on what the user selects as input. In somecases, users may choose public data and in other cases they may log intotheir personal account and use private data.

In a related embodiment, the inspiration source 710 can include aplurality of images 712.

In a related embodiment, as shown in FIG. 8, the product editor 214 canbe configured to provide a graphical user interface 800 for creatingconfigurations (mixing configurations), reviewing the results, andchanging input, and can include:

-   -   a) Creating configurations, wherein the machine learner 314 can        be configured to create configurations 810 using deep learning        artificial intelligence algorithms by placing materials from the        selected collection and the input from the inspiration source        710 on the selected three-dimensional object 400 c 510. The        machine learner 314 can be configured to process colors and        color combinations, and identify similar objects and analyze        colors to select materials and colors and assign them to the        individual layers/portions 410 of the selected 3D object 400 c        510. In addition, materials can be selected and assigned based        on their area of use, such as rubber for soles, liner. Material        and color assignments may result in new combinations that are        not yet available as a material. The new combinations may be        submitted as new material requests to the suppliers directly        from within the product configuration design system 100.    -   b) Reviewing the results, wherein the product visualizer 210 can        be configured to display the resulting configurations, for        example in a grid of 4 by 4 configurations. The machine learner        314 can execute a remixing of the content to show sixteen new        configurations.        -   All configurations are being generated locally on the            product configuration design device 104. The results are            being displayed in a grid form (16, 4, 21, 8), showing the            detail of each configuration including visual representation            and other relevant information if desired, such as overall            material cost, for example. Each configuration can be opened            in the interactive 3D real-time window for in depth review            as shown further in FIG. 11. Additional information that            will be available for each configuration is a detailed bill            of materials. More configurations can be available by            scrolling to the next page.    -   c) Changing input, wherein the product editor 214 can be        configured to allow the user 122 to go back and revisit any of        the 3 selections made in previous application steps        corresponding to views 500, 600, and 700, by simply clicking on        the corresponding icon in the bottom toolbar.        -   At any point in time results can be recalculated by changing            the input parameters. By clicking onto the icons on the            bottom tool bar the user can step back and change the            original 3D object 400 c 510, or the collection 610, or the            inspiration source 710.

In a related embodiment, each configuration in the plurality ofconfigurations can include, the three-dimensional object representation400 c, which comprises a plurality of regions 410, such that for eachregion 410 a corresponding material representation 400 b with acorresponding color combination is applied.

In a related embodiment, as shown in FIG. 9, the product editor 214 canbe configured to provide a graphical user interface 900 for acceptingand rejecting of results, remixing, polling, saving, such that:

-   -   a) Accepting or rejecting of results, wherein the product editor        214 can allow the user 122 to accept or reject result        configurations through either left (accept) or right (reject)        swipe.        -   In order to accept or reject results, the user either swipes            left or right. Accepted and rejected combination will be            used by the machine learner 314 for training a configuration            generation model in order to deliver better, more desirable            content in the future. Accepted configurations will stay as            part of the results, with a green indicator to denote that            the configuration is approved, while rejected results will            be removed and allow for the space to be filled by a new            configuration. Configurations can also not be accepted or            rejected, and will therefore not be considered as input for            Machine Learning. These unrated results will be overwritten            during the next creation of configurations (mix) but will            likely be saved as the AI/ML would likely not want to            represent them again. This will move the invention away from            a complete randomizer to a more intelligent system.    -   b) Remixing, wherein, as shown in FIG. 9, the product editor 214        can also enable the user 122 to add a new configuration 810 to        replace a rejected configuration 810. It also gives an option to        restart the configuration based on the most recent input.        -   Remixing means a new creation of either an individual            configuration after a certain configuration has been            rejected. After rejection of a certain configuration, the            user can click on the icon in the empty space and create a            new configuration. A completely new creation of a mix is            also possible by clicking on the “mix” or “remix” icon in            the toolbar on the bottom.    -   c) Polling, wherein swiping from the bottom up will send a        configuration to a group of people for simple voting. All votes        will be taken into consideration based on which a final decision        can be made by the original author.        -   Users 122 can create a poll of a configuration by swiping up            on the result. This will create package that can be shared            with people inside or outside their organization to help            rate the configuration. This rating will happen on a mobile            device (or browser-based system or any electronic device) as            shown in FIG. 10. Once polling is finished, the user can            review poll results and then accept or reject the            configuration accordingly. All accepted or rejected result            including the actual polling results for a certain            configuration will be stored and used for Machine Learning.    -   d) Saving—Saving a configuration, invoked by swiping from the        top down, which will save the current configuration, and remove        it from the results        -   To save configurations the user can lock a certain            configuration by swiping from the top down on a single            result. Saving a configuration will also affect Machine            Learning, and can be weighted with a greater influence than            liked or unliked configurations.

In a related embodiment, the product configuration design device 104 canbe configured to display a plurality of configurations 810, such thatthe user 122 is enabled to accept or reject each configuration 810 inthe plurality of configurations 810, such that the user identifies aplurality of accepted configurations 810 and a plurality of rejectedconfigurations 810, which are associated with the three-dimensionalobject representation 400 c, the collection 610, and the inspirationsource 710.

In a related embodiment, as shown in FIG. 10, the product editor 214 canbe configured to provide a graphical user interface 1000 for acceptingand rejecting of results on a mobile phone. FIG. 10 illustrates themobile phone application which allows users to review and vote onconfigurations that have been shared with them.

In a further related embodiment, to gain better insight on popularityand to capture third party feedback configurations, before they havebeen accepted or rejected by the user, configurations 810 can be sharedwith people inside or outside their organization. In FIG. 10 an exampleof an application view (which can be part of a browser or an app) isshown that shares a certain configuration with the associated 3D model.The user/recipient 122 can be able to review the 3D model with theconfiguration in an interactive 3D window. Alternatively, the recipientwill also be able to click through saved views to quickly gain access todetails the user wants to be reviewed. Using the same principle from theoriginal application the recipient can accept or reject theconfiguration by swiping left or right (or tapping/selecting the acceptor reject buttons). This will contribute to the polling of theconfiguration inside the application and will also be used by themachine learner 314 for training the configuration generation model. Inaddition, additional data can be captured such as age, gender anddemographics to allow for more target driven design in the future.

In a related embodiment, as shown in FIG. 11, the product visualizer 210can be configured to provide a graphical user interface 1100 forreviewing result configurations 1110 in 3D, such that individual mixresult configurations can be viewed in a fully interactive, real-time 3Denvironment. The tags 1120 show exactly what sources of input lead tothe result. Any result can be reviewed in a 3D interactive real-timeviewer. Inside the viewer the user can spin and pan the photorealistic3D object in real-time, and zoom the camera closer to the object toreview details in close-up view. To explore the behavior of the materialand color configurations under different real-world lighting conditions,the user can swap out the image-based lighting environment. In addition,the user can also review the 3d object in real-world context using AR.

In a related embodiment, as shown in FIG. 12, the product visualizer 210can be configured to provide a graphical user interface 1200 formanipulating the individual result configurations 1110. FIG. 12 showsthe user interface for manual interaction with the 3D object to eitherstart a new configuration or refine a suggested or existingconfiguration, thereby providing the user 122 an ability to manipulateand refine individual results. Users can choose to apply materials,colors and graphics manually to layers of the 3D object. Layers, alsocalled regions, can then also be locked so they don't get changed duringthe configuration process. All of this input will then be used as inputin combination with the inspiration input of various libraries to createnew configurations using materials and colors from the selectedcollection. These manual selections can be used by the machine learner314 for training the configuration generation model, thereby helping todefine the results of configurations in the future.

In a further related embodiment, the configuration generation model 316can be trained with the plurality of accepted configurations 810 and theplurality of rejected configurations 810, based on an input of thethree-dimensional object representation 400 c, the collection 610, andthe inspiration source 710, such that the configuration generation model316 is optimized to generate the accepted configurations 810.

In a further related embodiment, at least one accepted configuration 810in the plurality of accepted configurations 810 can include thethree-dimensional object representation 400 c, which comprises aplurality of regions 410, such that for each region a correspondingmaterial representation 400 b with a corresponding color combination isapplied, wherein the plurality of regions 410, comprises at least onelocked region 410, which is applied with a locked materialrepresentation 400 b with a locked color combination, such that theconfiguration generation model 316 is trained to only output productconfigurations 810 wherein the at least one locked region 410 isassociated with the locked material representation 400 b with the lockedcolor combination.

In a related embodiment, as shown in FIG. 13, the product visualizer 210can be configured to provide a graphical user interface 1300 forreviewing all individual configurations (mixes). The graphical userinterface 1300 illustrates a representation of all the savedconfigurations 1110. Any of the saved configurations 1110 can be openedand reviewed, with individual configuration to be opened in the 3Dreal-time interactive window for review, presentation and refinement.All results of various mixes are saved and stored locally and will besynced with the swatchbook database. Mixes can also be shared with otherpeople inside the organization for further review and collaboration.From the collection of mixes, the user can invoke actions directly onindividual mixes such as polling, sharing and collaborating.

Thus, in various related embodiments, the product configuration designsystem 100 can provide:

-   -   a) A new type of design tool that combines a traditional design        configuration tool with artificial intelligence (AI) and machine        learning (ML);    -   b) An AI engine that is feeding off a library of materials and        colors, and input from custom image collection stored in the        cloud, on a computer, or on social media;    -   c) A system that is taught by collected images, trends,        acceptance and rejections of certain color and material        combinations;    -   d) An automated configurator that applies materials and        combination of materials and colors using AI by looking at        images and collections on INSTAGRAM™, FACEBOOK™, PININTEREST™        and more;    -   e) An automated configurator that can be used in full or partial        automation mode by allowing the user to look down one or        multiple parts on the object;    -   f) A design tool that can be taught by using ML based on liking,        disliking, and saving material/color combinations, resulting in        more desired automated design combinations based on learned        selections;    -   g) An AI based material and color configurator using additional        parameters such as region, age, gender and more;    -   h) A smart way using AI of applying materials based on their        type and tags, and use on certain parts of object. (e.g. rubber        only goes on sole, no fabric on sole);    -   i) A new way of capturing trend data using ML;    -   j) Using a manual selection method of assigning and locking data        to further guide and train ML;    -   k) Sharing of configurations to validate selected combinations        with people outside the organization on a mobile device,        providing further input to ML; and    -   l) The potential use of guided ML vs directed ML vs freeform ML        or any and all combinations of such ML driven processes.

In related embodiments, the product visualizer 210 can be configured toallow a user to apply 2D images to 3D objects via a tagging system, inorder to interactively visualize the image in combination with simulatedreal-world materials under simulated lighting conditions. Thecombination of images and digital materials can be saved into an imagelibrary 212 for further inspiration, review, refinement, new materialdevelopment in cooperation with material suppliers. Furthermore, anymaterial stored in the online library can be viewed by simple additionof smart-tags that are associated with 3D models, and associated colorranges of the material can be visualized. The functionality providedincludes:

-   -   a) A tagging system that is fully customizable, allowing users        to define and set up context sensitive tags where a tag exposes        a relevant collection of sub-tags. The tagging system can have        an unbounded depth of sub-tags;    -   b) Tags that can be associated with a 3D object/model;    -   c) An application that allows users to visualize 2D images on 3D        models through simple tagging;    -   d) A way of visualizing color ranges of materials on a 3D model;    -   e) An interactive viewing application for mobile devices; and    -   f) An interactive view for new material development using mobile        devices.

In a related embodiment, the product visualizer 210 of the productconfiguration design device 104 can be configured to overlay a digitalmaterial representation 400 b of a physical sample 400 a onto thedigital product model/representation 400 c, in order to generate adigital product rendering 1010, as for example shown in FIG. 10, suchthat the digital product rendering 1010 can be viewed by the user 122.The product visualizer 210 can be configured to generate views withlighting shadowing and depth of field, to simulate the appearance of aphysical product.

In a related embodiment, as illustrated in FIG. 14, the productvisualizer 210 can be configured with an application view for smart tagcreation 1400, to allow a user to set up smart tags 1410 according to acustomizable tagging system. The customizable tagging system allows forcreation of tags, which are context sensitive, such that the taggingsystem allows for defining dependencies of tags, within a tagtaxonomy/hierarchy, such that tags can be parent tags, which can beassociated with specific sub-tags that apply to such particular parenttag. A sub-tag can belong to multiple parent tags. A sub-tag can act asa parent tag to other sub-tags. The dependencies, referred to as“depth”, can be unbounded, such that a user 122 can set up as manylevels of sub-tags as desired. A tagging system or subset of a taggingsystem can be set up for a manufacturer/make, and its associated brands,and models. A master/parent tag for a particular brand, can for examplehave a footwear sub-tag, which has the further sub-tags {athletic men,athletic women, casual men, casual women, formal men, formal woman},each of which has an associated set of model sub-tags, each of which areassociated with at least one 3D model.

In a related embodiment, a smart tag can be associated with a

-   -   a) numerical value;    -   b) numerical range;    -   c) text descriptor; and/or    -   d) a 3D object shape representation.

In a related embodiment, the 3D object 400 c may define/include separatematerial surfaces/regions 410 of the 3D object 400 c, which canrepresent separate fabric cuts, or areas to which the materialrepresentation 400 b can be applied. For a shoe, for example, certainsurfaces 410 may have a material 400 b, such as fabric or leatherapplied, while other areas, such as the sole may not have the material400 b applied.

In a related embodiment, as shown in FIG. 15, the product visualizer 210can be configured to show a result design configuration 1510 in anaugmented reality/virtual reality (AR/VR) environment 1520. The AR/VRenvironment can be accessed by clicking on an icon.

The interaction with the camera 207 can be specific to the AR/VRcontrols provided by a toolkit/API used to render the AR/VR environment.In AR/VR mode, users will still be able to edit the image on the object,as shown in FIGS. 11 and 12.

In another related embodiment, the customizable tagging system caninclude a hierarchy of tags, such that each tag in the hierarchy of tagsis associated with at least one three-dimensional object.

In yet a related embodiment, the customizable tagging system can includea hierarchy of tags, comprising at least one parent tag, which isassociated with a plurality of sub-tags.

In yet a related embodiment, the associated three-dimensional object caninclude a plurality of material surfaces, such that the two-dimensionalmaterial sample is applied solely to the plurality of material surfaces.

In a related embodiment, the two-dimensional material sample can berepeated in a tiled structure across the surfaces of the associatedthree-dimensional object.

In a related embodiment, the product visualizer can be configured toadjust a size of the two-dimensional material sample relative to thesurfaces of the associated three-dimensional object.

In a related embodiment, the product visualizer can be configured toadjust a position of the two-dimensional material sample relative to thesurfaces of the associated three-dimensional object.

In related embodiments, the product configuration design device 104 canbe deployed as a stand-alone app, or it can be provided as part of aplatform of design and manufacturing apps, such that the productconfiguration design system 100 is embedded within the design andmanufacturing platform. The design and manufacturing platform caninclude other apps, including a digital material design andvisualization platform with a material mix application, which can takeas input a plurality of parameters (or ingredients) for a material suchas base, color, pattern, coating, and develop the material on the flybased on the inputs. The material mix application can be used for newmaterial creation, such as for material types like leather, with colors,hole patterns, with rules, and with inspiration to create new materials.

In a further related embodiment, the product configuration design device104 can include or integrate with a dedicated rendering application forthe creation of photorealistic imagery for sales, marketing and retail;such that the rendering application is either included as internalcomponent or via integration with external rendering application orsystem.

Thus, in a yet further related embodiment, such a digital materialdesign and visualization platform including the product configurationdesign system 100 can function as an ingredient based digital mixingplatform to rapidly create variations based on rules, specifications andinspiration, wherein user voting and consumer voting feeds into amachine learning backend to guide the product creation and decisionmaking. The ingredient based digital mixing platform can further includea photography mix application, which allows a user to select a digitalasset (pre-configured), select product type, select inspiration, suchthat the photography mix application can provide lighting variations andcamera position variation, and then proceed to polling/voting.

In an embodiment, as illustrated in FIG. 16, a method of productconfiguration design 1600, can include:

-   -   a) Selecting a three-dimensional object 1602, including        processes as described and shown in relation to FIG. 5;    -   b) Selecting a collection 1604, including processes as described        and shown in relation to FIG. 6;    -   c) Selecting an inspiration source 1606, including processes as        described and shown in relation to FIG. 7;    -   d) Generating configurations 1608, including processes as        described and shown in relation to FIG. 8, wherein a plurality        of configurations are generated as an output from a machine        learning calculation on a configuration generation model, which        takes as input the three-dimensional object representation, the        collection, and the inspiration source;    -   e) Accepting/rejecting result configurations 1610, including        processes as described and shown in relation to FIGS. 9 and 10;    -   f) Reviewing result configurations 1612, including processes as        described and shown in relation to FIG. 11; and    -   g) Editing result configurations 1614, including processes as        described and shown in relation to FIG. 12.

In a related embodiment, accepting/rejecting result configurations 1610can further include displaying the plurality of configurations, whereina user accepts or rejects each configuration in the plurality ofconfigurations, such that the user identifies a plurality of acceptedconfigurations and a plurality of rejected configurations, which areassociated with the three-dimensional object representation, thecollection, and the inspiration source.

In a further related embodiment, the method of product configurationdesign 1600 can further include training 1616 the configurationgeneration model with the plurality of accepted configurations and theplurality of rejected configurations, based on an input of thethree-dimensional object representation, the collection, and theinspiration source, such that the configuration generation model isoptimized to generate the accepted configurations.

FIGS. 1, 2, 3, and 16 are block diagrams and flowcharts, methods,devices, systems, apparatuses, and computer program products accordingto various embodiments of the present invention. It shall be understoodthat each block or step of the block diagram, flowchart and control flowillustrations, and combinations of blocks in the block diagram,flowchart and control flow illustrations, can be implemented by computerprogram instructions or other means. Although computer programinstructions are discussed, an apparatus or system according to thepresent invention can include other means, such as hardware or somecombination of hardware and software, including one or more processorsor controllers, for performing the disclosed functions.

In this regard, FIGS. 1, 2, and 3 depict the computer devices of variousembodiments, each containing several of the key components of ageneral-purpose computer by which an embodiment of the present inventionmay be implemented. Those of ordinary skill in the art will appreciatethat a computer can include many components. However, it is notnecessary that all of these generally conventional components be shownin order to disclose an illustrative embodiment for practicing theinvention. The general-purpose computer can include a processing unitand a system memory, which may include various forms of non-transitorystorage media such as random-access memory (RAM) and read-only memory(ROM). The computer also may include nonvolatile storage memory, such asa hard disk drive, where additional data can be stored.

FIG. 1 shows a depiction of an embodiment of the product configurationdesign system 100, including the product configuration design server102, and the product configuration design device 104. In this relation,a server shall be understood to represent a general computing capabilitythat can be physically manifested as one, two, or a plurality ofindividual physical computing devices, located at one or severalphysical locations. A server can for example be manifested as a sharedcomputational use of one single desktop computer, a dedicated server, acluster of rack-mounted physical servers, a datacenter, or network ofdatacenters, each such datacenter containing a plurality of physicalservers, or a computing cloud, such as AMAZON EC2™ or MICROSOFT AZURE™.

It shall be understood that the above-mentioned components of theproduct configuration design server 102 and the product configurationdesign device 104 are to be interpreted in the most general manner.

For example, the processors 202 302 can each respectively include asingle physical microprocessor or microcontroller, a cluster ofprocessors, a datacenter or a cluster of datacenters, a computing cloudservice, and the like.

In a further example, the non-transitory memory 204 and thenon-transitory memory 304 can each respectively include various forms ofnon-transitory storage media, including random access memory and otherforms of dynamic storage, and hard disks, hard disk clusters, cloudstorage services, and other forms of long-term storage. Similarly, theinput/output 206 and the input/output 306 can each respectively includea plurality of well-known input/output devices, such as screens,keyboards, pointing devices, motion trackers, communication ports, andso forth.

Furthermore, it shall be understood that the product configurationdesign server 102 and the product configuration design device 104 caneach respectively include a number of other components that are wellknown in the art of general computer devices, and therefore shall not befurther described herein. This can include system access to commonfunctions and hardware, such as for example via operating system layerssuch as WINDOWS™, LINUX™, and similar operating system software, but canalso include configurations wherein application services are executingdirectly on server hardware or via a hardware abstraction layer otherthan a complete operating system.

An embodiment of the present invention can also include one or moreinput or output components, such as a mouse, keyboard, monitor, and thelike. A display can be provided for viewing text and graphical data, aswell as a user interface to allow a user to request specific operations.Furthermore, an embodiment of the present invention may be connected toone or more remote computers via a network interface. The connection maybe over a local area network (LAN) wide area network (WAN) and caninclude all of the necessary circuitry for such a connection.

In a related embodiment, the product configuration design device 104communicates with the product configuration design server 102 over anetwork 106, which can include the general Internet, a Wide Area Networkor a Local Area Network, or another form of communication network,transmitted on wired or wireless connections. Wireless networks can forexample include Ethernet, Wi-Fi, BLUETOOTH™, ZIGBEE™, and NFC. Thecommunication can be transferred via a secure, encrypted communicationprotocol.

Typically, computer program instructions may be loaded onto the computeror other general-purpose programmable machine to produce a specializedmachine, such that the instructions that execute on the computer orother programmable machine create means for implementing the functionsspecified in the block diagrams, schematic diagrams or flowcharts. Suchcomputer program instructions may also be stored in a computer-readablemedium that when loaded into a computer or other programmable machinecan direct the machine to function in a particular manner, such that theinstructions stored in the computer-readable medium produce an articleof manufacture including instruction means that implement the functionspecified in the block diagrams, schematic diagrams or flowcharts.

In addition, the computer program instructions may be loaded into acomputer or other programmable machine to cause a series of operationalsteps to be performed by the computer or other programmable machine toproduce a computer-implemented process, such that the instructions thatexecute on the computer or other programmable machine provide steps forimplementing the functions specified in the block diagram, schematicdiagram, flowchart block or step.

Accordingly, blocks or steps of the block diagram, flowchart or controlflow illustrations support combinations of means for performing thespecified functions, combinations of steps for performing the specifiedfunctions and program instruction means for performing the specifiedfunctions. It will also be understood that each block or step of theblock diagrams, schematic diagrams or flowcharts, as well ascombinations of blocks or steps, can be implemented by special purposehardware-based computer systems, or combinations of special purposehardware and computer instructions, that perform the specified functionsor steps.

As an example, provided for purposes of illustration only, a data inputsoftware tool of a search engine application can be a representativemeans for receiving a query including one or more search terms. Similarsoftware tools of applications, or implementations of embodiments of thepresent invention, can be means for performing the specified functions.For example, an embodiment of the present invention may include computersoftware for interfacing a processing element with a user-controlledinput device, such as a mouse, keyboard, touch screen display, scanner,or the like. Similarly, an output of an embodiment of the presentinvention may include, for example, a combination of display software,video card hardware, and display hardware. A processing element mayinclude, for example, a controller or microprocessor, such as a centralprocessing unit (CPU), arithmetic logic unit (ALU), or control unit.

Here has thus been described a multitude of embodiments of the productconfiguration design system 100, the product configuration design device104, and methods related thereto, which can be employed in numerousmodes of usage.

The many features and advantages of the invention are apparent from thedetailed specification, and thus, it is intended by the appended claimsto cover all such features and advantages of the invention, which fallwithin the true spirit and scope of the invention.

For example, alternative embodiments can reconfigure or combine thecomponents of the product configuration design server 102 and theproduct configuration design device 104. The components of the productconfiguration design server 102 can be distributed over a plurality ofphysical, logical, or virtual servers. Parts or all of the components ofthe product configuration design device 104 can be configured to operatein the product configuration design server 102, whereby the productconfiguration design device 104 for example can function as a thinclient, performing only graphical user interface presentation andinput/output functions. Alternatively, parts or all of the components ofthe product configuration design server 102 can be configured to operatein the product configuration design device 104.

Many such alternative configurations are readily apparent and should beconsidered fully included in this specification and the claims appendedhereto. Accordingly, since numerous modifications and variations willreadily occur to those skilled in the art, the invention is not limitedto the exact construction and operation illustrated and described, andthus, all suitable modifications and equivalents may be resorted to,falling within the scope of the invention.

What is claimed is:
 1. A product configuration design system,comprising: a) a product configuration design server, comprising aconfiguration generation model; and b) a product configuration designdevice, such that the product configuration design device is connectedto the product configuration design server; wherein the productconfiguration design device is configured to enable a user to select athree-dimensional object representation, a collection, and aninspiration source; such that the product configuration design servergenerates a plurality of product configurations as an output from amachine learning calculation on the configuration generation model,which takes as input the three-dimensional object representation, thecollection, and the inspiration source.
 2. The product configurationdesign system of claim 1, wherein the collection comprises a pluralityof materials and a plurality of colors.
 3. The product configurationdesign system of claim 1, wherein the inspiration source comprises aplurality of images.
 4. The product configuration design system of claim1, wherein each product configuration in the plurality of productconfigurations comprises the three-dimensional object representation,which comprises a plurality of regions, such that for each region acorresponding material representation with a corresponding colorcombination is applied.
 5. The product configuration design system ofclaim 1, wherein the product configuration design server comprises: a) aprocessor; b) a non-transitory memory; c) an input/output component; andd) a machine learner, which is configured to process a machine learningalgorithm for training and executing the configuration generation model;all connected via e) a data bus.
 6. The product configuration designsystem of claim 5, wherein the configuration generation model is aconvolutional artificial neural network with at least two hidden layers.7. The product configuration design system of claim 5, wherein theproduct configuration design server further comprises: a productstorage, for storing a customizable tagging system, comprising ahierarchy of tags, such that each tag is associated with at least onethree-dimensional object.
 8. The product configuration design system ofclaim 5, wherein the product configuration design device is configuredto display the plurality of product configurations, such that the useris enabled to accept or reject each product configuration in theplurality of product configurations, such that the user identifies aplurality of accepted configurations and a plurality of rejectedconfigurations, which are associated with the three-dimensional objectrepresentation, the collection, and the inspiration source.
 9. Theproduct configuration design system of claim 8, wherein theconfiguration generation model is trained with the plurality of acceptedconfigurations and the plurality of rejected configurations, based on aninput of the three-dimensional object representation, the collection,and the inspiration source, such that the configuration generation modelis optimized to generate the accepted configurations.
 10. The productconfiguration design system of claim 9, wherein at least one acceptedconfiguration in the plurality of accepted configurations comprises thethree-dimensional object representation, which comprises a plurality ofregions, such that for each region a corresponding materialrepresentation with a corresponding color combination is applied,wherein the plurality of regions, comprises at least one locked region,which is applied with a locked material representation with a lockedcolor combination, such that the configuration generation model istrained to only output product configurations wherein the at least onelocked region is associated with the locked material representation withthe locked color combination.
 11. A method of product configurationdesign, comprising: a) selecting a three-dimensional objectrepresentation; b) selecting a collection; c) selecting an inspirationsource; and d) generating a plurality of product configurations, whereinthe plurality of product configurations are generated as an output froma machine learning calculation on a configuration generation model,which takes as input the three-dimensional object representation, thecollection, and the inspiration source.
 12. The method of productconfiguration design of claim 11, wherein the collection comprises aplurality of materials and a plurality of colors.
 13. The method ofproduct configuration design of claim 11, wherein the inspiration sourcecomprises a plurality of images.
 14. The method of product configurationdesign of claim 11, wherein each product configuration in the pluralityof product configurations comprises the three-dimensional objectrepresentation, which comprises a plurality of regions, such that foreach region a corresponding material representation with a correspondingcolor combination is applied.
 15. The method of product configurationdesign of claim 11, wherein the configuration generation model is aconvolutional artificial neural network with at least two hidden layers.16. The method of product configuration design of claim 11, furthercomprising displaying the plurality of product configurations, wherein auser accepts or rejects each product configuration in the plurality ofproduct configurations, such that the user identifies a plurality ofaccepted configurations and a plurality of rejected configurations,which are associated with the three-dimensional object representation,the collection, and the inspiration source.
 17. The method of productconfiguration design of claim 16, further comprising training theconfiguration generation model with the plurality of acceptedconfigurations and the plurality of rejected configurations, based on aninput of the three-dimensional object representation, the collection,and the inspiration source, such that the configuration generation modelis optimized to generate the accepted configurations.
 18. The method ofproduct configuration design of claim 17, wherein at least one acceptedconfiguration in the plurality of accepted configurations comprises thethree-dimensional object representation, which comprises a plurality ofregions, such that for each region a corresponding materialrepresentation with a corresponding color combination is applied,wherein the plurality of regions, comprises at least one locked region,which is applied with a locked material representation with a lockedcolor combination, such that the configuration generation model istrained to only output product configurations wherein the at least onelocked region is associated with the locked material representation withthe locked color combination.